Traffic monitoring is a vital element of network and system management. Traffic monitoring used to be a relatively straightforward task. In the past, many machines were connected to a single shared network, and a single instrument connected to the network could monitor all of the traffic. Requirements for increased bandwidth, changes in traffic patterns, and the quickly falling price of packet switching and routing devices, however, has caused a rapid movement away from shared networks to networks that are highly segmented. The challenge is to monitor traffic on these segmented networks.
One measurement that has become vital to network monitoring is the volume of traffic exchanged between nodes in a network. Such measurements are used for a wide variety of applications, including capacity planning, congestion monitoring, security analysis, and accounting/billing. For any given network, these measurements can be taken over every node permutation on the network to create a traffic matrix. For example, FIG. 1 illustrates a simple network 10 that contains nodes A, B, and C, which are capable of communicating with each other through any combination of Routers R1, R2, R3, and R4. As can be seen in FIG. 2, a traffic matrix 12 containing traffic flow counts is provided, each of which is associated with a unique pair of nodes. For example, a unique node pair can include nodes A and B, with node A being a source node and node B being a destination node (in which case, the traffic flow count would be 100, as shown in the example of FIG. 2), or nodes B and A, with node B being a source node and node A being a destination node (in which case, the traffic flow count is 64, as shown in the example of FIG. 2). Notably, it is a possible for a particular node to communicate with or through itself, e.g., when the node represents a subset of end nodes. For example, a unique node pair can include node A as being both a source node and a destination node (in which case, the traffic flow count is 10, as shown in the example of FIG. 2).
Although, in theory, the generation of a network traffic matrix is simple, its practical implementation is difficult to accomplish in an accurate manner. For example, as shown in FIG. 3, a data collector 14 can collect data from a single monitoring point located at Router 1 adjacent to node A. Since the monitoring point at Router 1 observes all of the traffic in and out of node A, the number of data packets transmitted from node A to nodes B and C, and the number of data packets transmitted from nodes B and C to node A, can be monitored. Because the monitoring point at Router 1 seldom or never monitors the data traffic in and out of nodes B and C, however, the number of data packets transmitted between nodes B and C is not known, resulting in the incomplete traffic matrix 16 illustrated in FIG. 4.
The number of monitoring points can be increased, so that more of the data traffic between the nodes can be monitored. For example, as shown in FIG. 5, the data collector 14 can collect data from multiple monitoring points respectively located at Routers R1 and R3 adjacent nodes A and B. In this manner, all traffic that involves nodes A and B will be monitored at least once. Notably, traffic flowing from node C back to itself will still not be monitored in the arrangement in FIG. 5. Because traffic transmitted between a particular pair of nodes may be seen by more than one monitoring point, care must be taken to avoid double counting when combining results at the data collector 14. For example, as shown in the traffic matrix 18 illustrated in FIG. 6, data packets flowing between nodes A and B will be counted twice. Although double-counting is easy enough to account for in simple networks with very few nodes, in large networks with many thousands of nodes and a large number of monitoring points, it can be very difficult to determine how many times a particular flow of packets has been counted—especially in a dynamically changing network.
If it is assumed that for each pair of nodes there is at least one monitoring point that can see and count all the data packets between them, then duplicate counts can be resolved by the data collector 14 by using the maximum of the data packet counts received from the multiple monitoring points for any given traffic flow (e.g., from node A to node B), while ignoring any lesser data counts for the same flow. In a practical network implementation, however, not all packets that pass through a particular monitoring point are examined in detail, but rather they are sampled (for example, one in every thousand data packets that flow through a sampling point may be examined in detail). In such a scenario, because the traffic flow counts for any given flow are now necessarily expressed as estimates with a mean and a variance, taking the maximum of the data counts will result in an upward bias in the estimated traffic flow count for any flow that was seen by multiple monitoring points.
There thus remains a need to provide an improved method and system for generating traffic matrices in data networks that sample data packets.